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ABSTRACT 

Career-technical educators face three issues in credentialing 
through assessment. First, the occupational credentialing domain is large and 
evolving, and a clear understanding of it is a prerequisite to considering 
adoption of a credential. Three types of credentialing are registration, 
certification, and licensure. Credentialing organizations are categorized by 
their mission (government -regulatory board, trade association, vendor- 
specific, National Skills Standards Board) . Oversight organizations are 
professional organizations that disseminate information and provide voluntary 
oversight by evaluating credentialing systems. Second, a set of clear, 
comprehensive standards is needed to define credential quality and 
credibility. Proposed evaluative criteria/standards to select assessment- 
credentialing are marketability, recognition, alignment to curriculum, 
quality of input standards, quality of assessments, and usability for career- 
technical education (CTE) setting. Third, CTE policymakers and educators need 
a rational, efficient process to evaluate credential systems and associated 
assessments against a set of standards. The following nine steps are the 
process: (1) define purposes and uses of occupational credentialing systems; 

(2) set evaluation criteria; (3) identify credentialing systems and evaluate 
preliminary link to programs; (4) conduct initial screening; (5) determine 
quality of input standards; (6) determine quality of credentialing 
assessments; (7) conduct final linkage to curriculum; (8) determine 
marketability and recognition; and (9) develop data collection procedures. 
(Contains a 53-item bibliography. ) (YLB) 
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Introduction 



In this paper, occupational or industry-based credentialing is considered from the perspective 
of the policymaker in career-technical education (CTE). A credential is awarded to an individual who 
meets predetermined qualifications maintained by a credentialing body (Browning, Bugbee & Mullins, 
1996; Schmitt, 1995; Shimberg, 1981). The qualifications usually are a combination of education, 
experience, and test scores. Ananda and Rabinowitz (1995) and Losh (2000) have noted that 
standards and occupational credentials can be important features of the CTE domain. Policymakers 
considering the adoption of credentialing might view the intended consequences as increasing access to 
occupations, signaling competence on the part of graduates, and demonstrating responsiveness of the 
educational system. Briefly, providing access refers to developing an alternative route by which 
individuals can enter the labor market, signaling competence refers to documenting capabilities of 
CTE graduates, and demonstrating responsiveness refers to signaling at the level of the educational or 
employment system. On the other hand, there may be unintended and less desirable consequences of 
adopting credentials. 

A selected set of examples helps to triangulate the current importance of credentialing. At the 
national level, the National Skill Standards Board (NSSB) is a strong advocate of certification as an 
integral part of its skill standards initiatives. In various publications, the NSSB has advocated bottom- 
up development of assessment-credentialing systems by voluntary partnerships (NSSB, 2000b, 

2001b). At the state level, California awarded multiple grants during 2002 to Local Education 
Authorities for implementing industry-based credentials in career technical education programs for high 
school students. Other states, Pennsylvania and Virginia for example, systematically identify and screen 
industry credentials. At the credential-provider level, students in CTE programs may seek credentials 
from a range of organizations. Large-scale credentialing systems, for example the Automotive Service 
Excellence (ASE) designation, are used in conjunction with automotive mechanic and collision 
technician programs. Another large domain of application is the information technology sector, in which 
vendor-specific networking or database administration credentials abound. 

Career-technical educators face three important issues in credentialing through assessment. 

First, the occupational credentialing domain is large and evolving, and a clear understanding of the 
domain is a prerequisite to considering adoption of a credential. Second, a set of clear and 
comprehensive standards is needed to define the quality and credibility of credentials. Third, career- 
technical policymakers and educators need a rational and efficient process to evaluate credential 
systems and associated assessments against a set of standards. In this review, the credentialing domain 
is described, a set of standards for evaluating the quality of a credential is proposed, and a collaborative 
process for evaluating credentials for CTE programs is illustrated The evaluation process relies heavily 
upon analysis of intended, as well as unintended, consequences of use within the CTE system. 
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Credentialing: A Large and Evolving Domain 



In this section, an overview of the credentialing domain is provided. A comprehensive 
treatment of the domain may be found in the edited book by Schoon and Smith (2000). First, three 
types of credentialing — registration, certification, and licensing — are defined, ordered from least to most 
restrictive. As part of this description, an overlap between certification and licensing is noted. Next, 
credentialing organizations are categorized Finally, oversight organizations in the credentialing domain 
are described; these professional organizations disseminate information and provide voluntary oversight 
by evaluating credentialing systems. 

The observation that the credentialing domain is large and evolving parallels one offered thirty 
years ago by Shimberg, Esser, and Kruger (1973). The size stems from the large number of 
organizations, varying in size and orientation, that offer credentials. For example, Microsoft and Cisco 
are large business firms within the information technology sector. The National Automotive Technical 
Education Foundation (NATEF) and the National Retail Federation (NRF) are industry-based 
foundations. The American Association of Occupational Health Nurses (AAOHN) typifies a 
professional association. The evolutionary change stems from the emergence of new credentials and 
from revisions of credential content standards and assessments. The research and practice knowledge 
base of the field is also expanding (Impara, 1995; Bugbee et al., 1996; Haladyna, 1997). 

Types of Credentialing 

Occupational credentialing include three specific types: registration, certification, and licensure. 
In this paper, the term “credentialing” is used to refer to the entire domain and “certification” is used 
only for the specific type of credentialing (as opposed to the more common usage of “certification” for 
both the domain and the specific type) in order to avoid confusion. Although the three types are distinct 
in theory, in practice there may be overlap between certification and licensure. 

Registration. Registration is the least restrictive form of regulation, usually requiring individuals 
to file their names, addresses, and qualifications with a government agency before practicing then- 
occupation. It may include posting a bond or filing a fee. Registration provides a ‘title’ more than 
anything else because individuals do not have to pass examinations to be registered. 

Certification. Certification is sometimes referred to as “title control”. It may be offered by a 
governmental body or by a profession (non-govemmental organization, or NGO) that gives right-to-title 
to persons meeting predetermined standards. Candidates usually must pass a certification exam. 
Certifications are granted to individuals who have met predetermined qualifications set by the 
certification agency within a profession. A certification is a formal recognition of professional or 
technical competence. Certification often may have leg^l implications for the certifying organization. 
Certification is divided into two broad domains: minimum competency standards and advanced 
knowledge standards. Minimum competency standards are used to represent what is deemed 
acceptable for an entry-level practitioner. Examples include Peace Officer certification, or Certified 
Public Accountant. Advanced knowledge standards are used to signal advanced knowledge and skill 
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within a profession. These are often specialization areas within occupations, for example cardiologists 
in medicine, tax specialists in accounting, or network administrators in information technology. 

Licensure. The term licensure is often referred to as “practice control” — a person cannot 
legally perform the occupation without the appropriate license. The primary purpose of licensure is the 
protection of the public. It is the most restrictive form of regulation imposed by the government 
agencies, and is often referred to as the “right-to-practice.” It is illegal under licensure laws for a 
person to practice a profession without first meeting state or provincial standards. Licensure is 
mandatory (in order to engage in the specified set of activities). Licensing is a process by which an 
agency of government grants permission to an individual to engage in a given occupation upon finding 
that the applicant has attained the minimal degree of competency required to ensure that the public 
health, safety, and welfare will be reasonably well protected. Occupational and professional licensure 
is an activity reserved to each state by the federal constitution; and involves the exercise of the state's 
legitimate policing power. The essence of licensure seems to be “protection of the public” although 
there are critiques that suggest an underlying desire to protect territory and exclude others from 
practice. 

By 1 952, more than 80 separate occupations had been licensed by state law. By 1 986, at least 
800 professions were licensed. Collectively regulated licenses exceeded 1000 professions in 1990 and 
continue to increase (Impara, 1995). For approximately 60 occupations (e.g., medicine, nursing, 
engineering) comparable licensing requirements exist in all states, thus establishing a national umbrella. 
For most professions, however, the requirements vary from state to state. While protection of the 
public is a core reason for licensure, clearly professional groups have stretched this meaning to attain 
multiple ends. The literature identifies these ends as including (1) enhanced economic benefits for 
practitioners, (2) increased status, (3) protection of the reputation of the profession, and (4) symbolic 
respectability. Last, licensure supports payment of services by third-party payers (i.e., insurance 
companies) and control over the number and geographic distribution of practitioners. 

Issues in types and vocabulary. Several issues are implied in these definitions. One issue is 
overlap. In general terms, registration, certification, and licensure are distinct types of occupational 
credentialing, but in practice there is overlap between them. They are not exclusive, especially in the 
healthcare field. The Ohio Peace Officer Certification, for example, is required by law for an individual 
to seek employment as a police officer in Ohio, but it is called a certification rather than a license. 
According to Schmitt and Shimberg (1996) distinctions between licensure and certification became less 
obvious in the 1970s, as many allied health groups chose to forego licensure and concentrate upon 
certification. A second issue is continuing competence. Change in occupations, together with lifelong 
learning, suggest that continuing competence must be addressed. (Tamblyn, 1994). A third issue is the 
legal implications for the credential provider that are associated with the various levels. These liabilities 
arise from the consequences of error — either error in awarding a credential to a candidate who is not 
qualified or error denying a credential to someone who is qualified. 
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Credentialing Organizations 

Organizations offering credentials fit into broad categories described by their mission. These 
missions, which maybe mixed, cluster into professional/public protection and profit motives. Table 1 
below describes several different credentialing organizations. As noted by Mahlman, Jeong, and Austin 
(2000), these include government-granted certifications and licenses and non-govemmental 
certifications offered by trade association or by proprietary vendors. The last row shows the generic 
CorePlus model advocated by the NSSB (2001b). This inclusion is warranted, we believe, because of 
the contributions of the NSSB. These include a common language for skill standards integrating work- 
and worker-oriented elements and creation of the voluntary partnership process. The authors 
encourage consideration of the implications of skill standards and certification for bridging CTE and 
workforce development (Ananda & Rabinowitz, 1995; Losh, 2000; National Skill Standards and 
Assessment Collaborative, 1998). 



Table 1: Selected Credentialing Organizations 


Type 


Example 


Credential 


Government 

Regulatory 

Board 


Cosmetology 

www.state.oh.us/cos/ 


State license (i.e., Ohio) based on 
a multi-part National Collaborative 
assessment 


Trade 

Association 


National Automotive Technical Education 
Foundation (NATEF) 


Automotive Service Excellence 
(ASE) certification, based on 
assessment scores and 
performance standards 


Vendor- 

Specific 


Microsoft Office User Specialist (MOUS) 
www. nr icrosoft.com/tra i ncert/ mcp/nious/ 


Microsoft Official Certificate with 
successful test score 


NSSB 


CorePlus “Model” 


Proposal to document core 
standards plus one specialization 



Oversight Organizations 

Two voluntary oversight organizations focus on certification, licensure, and associated issues. 
One is the Council on Licensure, Enforcement, and Regulation (CLEAR) (www.clearhq.oru) and the 
second is the National Organization for Competency Assurance (NOCA; www.noca.org ). CLEAR 
was established in 1980 and has evolved into an international “community” for those involved in 
professional and occupational regulation. NOCA was established in 1977 to focus on setting quality 
standards for credentialing organizations through its National Commission on Certifying Agencies, and 
those standards were recently revised (NCCA, 2002). Each group holds annual meetings and 
conferences, provides technical assistance, and serves as a clearinghouse. Membership includes a wide 
range of representation, including individuals as well as credentialing organizations (state regulatory 
boards, private providers, and professional associations). Both oversight organizations disseminate 

- 4 - 



6 



BEST COPY AVAILABLE 



information, including the NOCA handbook on certification (Browning, Bugbee, & Mullins, 1996), 
guidance for establishing cutoff scores (NOCA, 1999), issues involved in certification testing via 
Internet (NOCA, 2001), and explanatory material on the nuts and bolts of certification examinations for 
board members (CLEAR, 1993). The two organizations collaborated on Principles of Fairness 
(CLEAR & NOCA, 1993), which is organized around activities before, during, and after a credentialing 
test In summary, one implication of these organizations is that the credibility of a credentialing 
organization is enhanced if it belongs to one or both of these associations. 

Quality Standards for Credentials 

Occupational credentialing in principle offers CTE stakeholders several benefits. Stakeholder 
groups include students, parents, educational systems, businesses, and government agencies. Some 
potential benefits for students include increased marketability, greater respect from peers and 
management, increased job opportunity, more opportunities for advancement, and greater self 
confidence. The adoption of industry-based credentials also has the potential to offer benefits to 
educational systems. Good credentialing systems help CTE educators to document student 
competence. Furthermore, positive results or trends in results on credentialing exams could add 
considerable credibility to career-technical programs for many stakeholders. Business firms would gain 
if the assessment and credentialing system provided clearer signals about what individuals could do 
(Bishop, 1996; Rosenbaum, 2002; Zemsky, 1997). While the potential benefits are great, there are 
many industry credentials and all are not created equal. 

Credentials vary on many dimensions, including marketability and recognition of the credential, 
alignment of the underlying standards to the CTE curriculum, quality of the input standards used to 
create the test plan and items, technical quality of the assessment system (procedures, tests, and items), 
and usability in educational settings. The selection of credentials for CTE students must be performed 
carefully and systematically in order for students to realize the potential benefits. Selection of poor 
credentialing systems will likely perform a disservice to students and other stakeholders in the 
educational credentialing system. Some of the unintended consequences to stakeholders are negative 
publicity if scores appear low or declining, teacher-administrator conflict over apparently poor student 
performance, misrepresentation of student/worker competence, wasted time and financial resources, 
and reduced motivation for students. 

Before the selection of industry-based student credentialing systems can occur, CTE 
policymakers must define the purposes for adopting the credential and make some key decisions. Key 
decisions include at least the following: 

• Will the credential standards or curriculum need to fit with current state standards or 
curriculum? 

• Will the credentialing assessment system be used for state and/or federal accountability 
purposes or for program improvement purposes? 

• Will the credentialing system be used to market CTE graduates and/or programs? 
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Depending on the answer to these and similar questions, criteria or standards for selecting industry- 
based student credentialing systems and the relative emphasis on each will vary. 

Given these sorts of questions, how should policymakers think about assessment-credentialing 
systems and evaluate competing systems? The answer involves evaluative criteria that can be applied 
to evaluate existing certifications or to develop novel certifications. This section proposes the standards 
defined in Table 2 below. These specific standards are ones that could potentially differentiate 
competing systems on quality. The standards are then integrated into a recommended process for 
selecting assessment-credentialing systems. 



Table 2. Evaluative Criteria for Assessment-Credentialing Systems 


Criterion 


Clarifications/ Assumptions 


Marketability 


If related to an increased preference in hiring and an increase in wages, 
marketability is greater. 


Recognition 


If accepted by marry hiring organizations across a wide geographical 
area, recognition is greater. 


Alignment to Curriculum 


If closer match between content measured by the credentialing test(s) 
and content of CTE curriculum presented, alignment is greater. 


Quality of Input Standards 


Appropriateness of the standards upon which the credential is based: 
How were they developed? Are they current? Validated? 


Quality of Assessments 


Appropriateness of assessments on which credentialing decisions rely. 
Are they reliable, valid, & fair? Evaluation against standards, for 
example the Standards for educational and psychological testing 
(American Educational Research Association [AERA], American 
Psychological Association [APA] & National Council on Measurement 
in Education [NCME], 1999), or the ETS Test Quality Review 


Usability for CTE Setting 


System features: assessment cost, timing, availability of test results to 
educators/administrators, and data format 



Marketability and Recognition 

If a credentialing system is evaluated on the basis of marketability and recognition, the extent to 
which benefits are realized by students and/or CTE programs is a consequence of using the system. 
This suggests consideration of both intended and unintended results of using an assessment. First, are 
the intended consequences of adopting the system actually occurring? One short term outcome, for 
example, is improved access and signaling. In the longer term, adoption predicts an increase in labor 
market efficiency. Second, there should be anticipation and minimization of unintended negatives. An 
example of an unintended negative consequence is lower pass rates than expected even though the 
CTE curriculum is aligned with the credential content domain (as in Automotive Teclnology and 
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Automotive Body Repair). The management of negative consequences might include monitoring the 
pass rates as a credential is rolled out to a state and crafting communications with stakeholders to 
provide a context for the pass rates and expectations for their improvement over time. Information on 
marketability and recognition of a credential may be obtained through such activities as: 

• Communication with industry associations other than the association offering certification 

• Interviews and surveys — for example, surveys of credentialed workers that investigate their 
perceptions of the value of the credential, or surveys of employers 

• Follow-ups of certified individuals to study their short-term and long-term uses of the credential 

Alignment with Curriculum 

Fairness in testing requires that students have been exposed to and have had an opportunity to 
learn the material being tested. Alignment is the technical term for how different systems match, and 
two aspects of it are important. The first is the extent of overlap between the curriculum presented and 
the credentialing assessment. What proportion of the curriculum is covered by the credential 
assessment system? If a credential covers only a small portion of the curriculum, then the program as a 
whole cannot be evaluated by credentialing rates. A deficiency in coverage may be addressed by using 
multiple credentials that provide adequate coverage when used in combination. Second, there is the 
extent to which a credential addresses content that is not covered in the curriculum. This goes back to 
fairness in testing. A students achievement should not be evaluated by a credentialing test covering 
material that a student has not had the opportunity to learn. Evaluations conducted using this type of 
data perform an extreme disservice to the students and the programs being evaluated. Studying the 
alignment of the credential content domain and the CTE curriculum might include the following activities: 

• Having a panel of experts (1) match the claimed credential content to the curriculum and (2) 
evaluate overlap and contamination (illustrated in the process used by the Virginia Department 
of Education to award “diploma seals” in CTE) 

• Having a panel of experts match the credential test items (or samples of test items) to the 
curriculum and rating percentage of overlap and contamination 



Quality of Input Standards 

Most credentials signal that an individual has attained a level of competence, defined as 
knowledge or skill, in a prescribed content area. One consideration in the choice of a system is 
appropriateness of the content standards upon which the credential is based. Are they current and 
validated? Has job, occupational, or role analysis been used to delineate the duties, tasks, and 
associated knowledge or skill needed to perform within an occupation? Job, occupational, or role 
analyses generally include interviews and focus groups with individuals performing the job, observations 
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of workers, and surveys to validate and weight the resulting descriptions of the job. The job and task 
analyses, or role delineation studies, define the content covered by a credentialing assessment and the 
relative emphasis placed on content areas. Four study descriptions provide illustration of the process. 
Cameron, Beemsterboer, Johnson, Mislevy, Steinberg, and Breyer (2000) completed a cognitive task 
analysis of dental hygenists. Niebuhr and Muenzen (2001) described a study of the practice of 
perianesthesia nursing in order to update two credentialing examination programs. McMillan, 
Heusinkveld, Chai, Murphy, and Huang (2002) presented a thorough discussion of a role delineation 
study and its use to revise the blueprint for the Oncology Certified Nurse examination. Raymond 
(2002) reviewed practice analysis for credentialing assessment organized around six questions, 
providing a state-of-the-art review from the perspective of a credentialing professional. 

Such analyses, whether of jobs, practice, or roles, form the foundation of assessments and 
training procedures. The appropriateness and the timeliness of the results are critical to the quality of 
the credentialing system. Examining the quality of input information (job analysis, content standards) 
might include the following procedures: 

• Examining documentation about procedures used to establish the content of the credentialing 
system. 

• Examining task analyses to determine appropriateness of weighting of content standards. 

• Verifying the quality of standards, as illustrated in parallel surveys of perceived importance of 
the standards within each of eight allied health areas by Lyons, Greening, and Robeson (2000). 

• Studying the test specifications derived from the input materials. 

Technical Quality of the Assessment System 

Test quality, often under the umbrella label of validity, is a function of the procedures used to 
develop the test and the psychometric properties of the test and test items. Most validation evidence is 
of the content nature, given that the purpose is to assess a candidate’s level of knowledge and/or skill 
(Shimberg, 1981; Mehrens, 1997b). There are several sources that provide professional guidance, 
both general (AERA, APA & NCME, 1999) and specific to the certification domain (Haladyna, 1997; 
Impara, 1995). Other factors in test quality might include the extent to which the test is perceived to 
(1) be secure, (2) be current, and (3) avoid poor wording, graphics, and grammatical problems. Test 
administration procedures can be reviewed for their appropriateness, their standardization, and their 
security. Determining the quality of an assessment involves studying not only test development and 
administration procedures but also the empirical quality of the resulting assessments. Were the test 
specifications created from the content standards based on a quality job analysis (Raymond, 2002; 
Smith & Hambleton, 1990)? Who were the item writers and what were their qualifications? How 
were pass/fail cutoff scores determined? What procedures were used to establish the reliability and 
validity of the assessments? What were the results of those studies of reliability and validity? Are the 
test items themselves clear and free from grammatical errors? Are the distractors for multiple-choice 
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items plausible yet always incorrect? Are the assessments appropriate for the target population? 
Appropriateness, within the context of CTE, means that there is a consideration of whether students will 
be able to meet all of the qualifications maintained by the certifying organization. For example, if there 
are experience requirements and a high school student will not have an opportunity during their CTE 
program of study (typically the last two years), then requiring a certification will be unfair. 

Evaluation of test quality derives from judgments by individuals with expertise in testing and 
psychometrics. Their evaluations can be supported by additional judgements about the test and test 
items from content experts, editors, and graphics specialists. Evaluating test quality might include the 
following activities: 

• Having testing expert or panel of experts examine assessment technical manuals. 

• Interviewing test developers via phone or email to fill-in missing information. 

• Collecting additional evidence of reliability, validity, and fairness, if insufficient evidence exists or 
to extend the validity of the assessment (e.g., judgment-based and/or empirical procedures such 
as administering assessments to estimate reliability, demonstrating convergent validity, and/or 
establishing criterion-related validity by correlating credentialing test scores with external 
criteria, as shown in the psychometric analysis of a Canadian provincial examination for 
massage therapists by Violato, Salami, and Muiznieks [2002]). 

• Having tests reviewed by content experts, or subject matter experts. 

• Reviewing the procedures used to establish performance standards (Cizek, 2001; Kane, 1994; 
Plake, 1998). 

• Having tests reviewed by proofreaders and graphics specialists. 

Usability in Educational Settings 

Consider a credential that is of high quality, valid for the proposed uses, recognized, and 
marketable — but is not usable for demonstrating student achievement or for collecting accountability 
data. Several features influence usability. Examples are cost and timing of the assessment, availability 
of test results to educational institutions, and data format. Depending on use of credentialing results, 
another feasibility issue might be the degree to which the results across various credentialing systems are 
comparable. The impact of cost would be influenced by the source of funds. Fees for credentialing 
examinations may exceed $100.00. Who pays? Possible sources of funding include the state 
department of education, the school, and/or the student. Issues of timing, resulting from experience 
requirements for a credential, are also critical to data collection. If a student is required to spend two 
years in an apprenticeship before he or she can complete a credentialing exam, outcome data (Pass, 
Fail) might be available too late for external reporting cycles or too late for timely program 
improvement. The availability of data to school districts for these purposes is critical. Can student test 
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results be obtained directly from the credentialing organization or must districts rely on voluntary 
self-report by students? In what format are data provided? Is the format congruent with existing 
reporting mechanisms? In developing a system for utilizing industry-based credentialing to demonstrate 
CTE student competence, procedures are needed to collect student data and to enter that data into 
some form of information management system with archival capabilities. Sample activities for evaluating 
usability or feasibility might include the following: 

• Establishing criteria with respect to cost, timing, data collection procedures, and format. 

• Screening available credentials with respect to the criteria. 

• Planning a system for collection of student data on credentialing results. 

A Collaborative Process for Evaluating Credentials 

Given the laige number of occupational credentials, is there an efficient way to apply these 
quality standards to evaluate potential credentials for CTE usage? First, screening credentialing systems 
and their associated assessments is not a novel idea. Showers (1999) uses the term “audits” to refer to 
such a review and multiple organizations offer such services We briefly note several existing systems to 
provide a context for our proposal. Examples include the standards of the National Commission for 
Certifying Agencies (2003), the test review process used internally by ETS (2002), and the models 
proposed by the NSSB, the Buros Institute for Assessment Outreach and Consultation (BLACO), and 
the American National Standards Institute. The accreditation of credentialing organizations or 
programs can yield great value to those responsible for selecting industry-based credentialing programs 
for use in CTE. By reviewing the accreditation standards, the consumer of credentials can reasonably 
assume that at least those aspects of the accredited credentialing program hold integrity. However, 
since these accreditations are voluntary, a drawback is that currently only a relative handful of 
credentialing programs are accredited. As an example, BLACO provides on their website a listing of 
credentialing organizations in information technology that have achieved accreditation based on a set of 
customized evaluative criteria lvvww.unl.edu/BIACO) . 

The process should begin with the least expensive and most efficient screens. An example 
would be document review for a certification system Documents reviewed might include marketing 
materials, informational brochures, and technical manuals for the assessment More labor intensive 
screens (panel reviews, tryouts) can be completed as a list of eligible systems is refined. Although 
details of the proposed review might be customized for districts and for states, an ongoing review would 
provide fairness to certification assessment vendors. The proposed 9-step process is presented as a 
flowchart below, and then each step is described in detail. 
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Selection of Credentialing Systems for CTE 







1. 



Define the purposes and uses of occupational credentialing systems. The first step is to 
define the purpose for adopting a credentialing system. This definition is important to evaluate 
whether the system is working. If multiple purposes are to be served, the relative importance of 
each purpose should be established. The uses and purposes of the certification system will 
affect the criteria that will be chosen as well as their relative weights. Among the reasons for 
adopting external credentialing for career-technical education students are the following: 

• Increasing marketability of students (preferences in hiring, wages, promotion). 

• Recognizing professionalism for students (possibly for teachers and programs). 

• Enhancing credibility of programs (marketability applied to the program level). 

• Documenting competency attainment for accountability or impact reporting purposes. 

• Documenting competency attainment for program review and improvement purposes. 

Further, key policy decisions will need to be made in conjunction with defining the purposes 
and uses of the industry-based credentialing system. For example, will the standards on which 
the assessments are based replace or complement the state’s current system of occupational 
standards, competencies, or curriculum? This decision will have implications for the degree to 
which the certification content must match current curriculum. Will the certification assessment 
system be used for state and/or federal reporting purposes? If so, assessment system quality 
and usability in educational settings become critical, and data collection procedures need to be 
established. 

2. Set evaluation criteria. Answers to policy questions influence the relative emphases placed 
on criteria for evaluating credentialing systems. Criteria must be based on the purposes and 
proposed uses of the credentialing system. Emphasis on criteria can be distributed across 
marketability or recognition, linkage to curriculum, quality of input standards, technical quality of 
the assessment system, and usability in educational settings. Relative weighting depends on 
purposes and uses of the system. Usability criteria should be established a priori and should be 
transparent, that is, publicly available. They could, for example, include cost, timing of the 
assessments, availability of data to educational systems, and the format of that data. Depending 
on the system used to examine linkage to the curriculum, one might determine acceptable 
percentage levels of curriculum overlap. Other criteria, for example level of marketability or 
psychometric quality, may be more judgmental and require refinement as one begins the 
process of examining credentialing systems. 

3. Identify credentialing systems and evaluate preliminary link to programs. Once criteria 
are set, credentialing systems need to be identified and initial links established between these 
systems and the state's CTE programs and curricula. The initial link to a program is a cursory 
review and match based on the title of the credential and any readily available description of the 
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credential. At this time, less attention is paid to the criteria because of the time required to 
complete a comprehensive review. However, many credentialing systems may be immediately 
maiked for elimination if it is apparent that the criteria are not met 

4. Conduct initial screening. This stage involves an examination of descriptions of the credential 
with attention being paid to its perceived linkage to the curriculum, how it fares against the 
usability criteria, and any other criteria that can be quickly assessed. The goal here is to screen 
as many credentials as possible by examining characteristics of the credential that are easily 
obtainable and easy to review. This task might involve searching the web and contacting 
credentialing organizations for information. A more thorough linkage to the curriculum will be 
conducted at a later time, after further screening. Despite the initial nature of this step, 
documentation of the screening process should be thorough. 

5. Determine the quality of the input standards . Before determining quality of credentialing 
assessments, one should examine the quality of their foundation. Was a thorough job or 
occupational analysis performed? What were the qualifications and backgrounds of the subject 
matter experts that delineated the duties, tasks, and knowledge and skill needed to perform the 
job? Was a task analysis performed? Was the geographical representation of the analysis 
appropriate for your population? How critical is the established content of the credential to 
performance of the job or features of the job being credentialed? 

6. Determine the quality of credentialing assessments. Credentials surviving the initial 
screening should be examined against psychometric quality criteria. Here, professional staff 
contact the credentialing organization to obtain technical manuals and any other information that 
will describe test development procedures, psychometric quality information, and validation 
procedures and results. This step may involve independent content and grammatical reviews of 
items. If the purposes of the credentialing system include using it for individual, state, and/or 
federal accountability purposes or other high-stakes purposes, then quality of the credentialing 
assessments becomes critical. 

7. Conduct final linkage to the curriculum. The most direct method of determining the overlap 
and contamination of the assessments with respect to the curriculum is to have a panel of 
subjectmatter experts link the actual test items to the curriculum. This linkage will allow a 
quantitative determination of how much of the curriculum is covered, how well it is sampled, 
and the degree to which content is measured that is not part of the curriculum. The most 
efficient way to perform this would be to have the credentialing organization perform this 
linkage on a preliminaiy basis, with verification by subjectmatter experts familiar with the 
curriculum. Note that linkage to the state-adopted standards or curriculum is not an issue when 
the industry-based credential will replace the state adopted standards or curriculum. However, 
determining the degree to which the standards or curriculum are too rigorous or not rigorous 
enough for the targeted educational program may be an issue. 
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Determine marketability and recognition. Determining marketability and recognition 
become important when one of the purposes of the system is to enhance the marketability of the 
student and/or the CTE program. The task of evaluating marketability and recognition in some 
cases wifl involve thorough investigative work and thus will be labor-intensive. For this reason, 
it is saved only for credentialing systems meeting all other criteria. This task will require 
interviews obtained from multiple sources and/or surveys of employers and employees. The 
purpose of this step is to determine how widely the credential is recognized and what 
employment benefits the student might experience by holding the credential. 

Develop data collection procedures. If the purposes of the credentialing system include use 
of scores for individual, state, and/or federal accountability or program improvement purposes, 
then data procedures must be put in place. Procedures must be developed to collect the data 
on credentialing results and attach that data to individual students in an information management 
system. This would involve discussions and arrangements with the credentialing agency to 
develop data sharing procedures. Timing of data sharing is a critical issue here, especially when 
the CTE system must have the data available for external state or federal reporting purposes. 
Also, when working with minors one must attend to Family Educational Right To Privacy Act 
(FERPA) issues as well as state laws regarding use of data. 



Conclusion 



In closing this review, we wish to emphasize the following points. First, the screening of 
systems for use in credentialing CTE students should be approached with diligence. The consequences 
for students, teachers, programs, employers, and other stakeholders of a poor credentialing system can 
be great. Through careful planning and the appropriate allocation of resources, the selection of quality 
industry-based credentials for documenting CTE knowledge and skill attainment can yield immense 
benefits for students and for educational systems. Second, a prominent role was given to the idea that 
both use and consequence of use are important desiderata in evaluation of credentialing system. 
Messick (1995) argued that the consequences of usage must be considered in both intended and 
unintended senses, although the addition of consequences remains controversial in validation theory 
(Mehrens, 1997a). If viewed as worthwhile, suggested strategies for addressing consequential validity 
include focus groups and other stakeholder panels (Lane & Stone, 2002). As an example of how 
consequences could be anticipated and managed in the context of CTE, it may be possible to work 
with the credentialing organization to establish customized performance standards (cutoff scores) for 
secondary CTE student The intent of such standards would be to recognize the differences in 
knowledge and/or skill between the emerging workforce and such populations as postsecondary 
students or current occupational practitioners. 

In developing this review, certain larger implications became apparent to the authors. Among 
them are two that warrant particular attention: use of credentialing-assessment for system reform and 
ensuring coverage of all important worker characteristics. First, the implication that occupational 
credentialing-assessment can play a role in systemic reform may be contradictory to a “back to basics” 
movement. The “back to basics” movement places increased emphasis on basic academics and 
decreased emphasis on career and workplace preparation. The proposed evaluative criteria and 
screening process for credentialing-assessment could be transported outside of the secondary domain, 
for example to the community college or proprietary school. Second, if credentials are to be used it is 
important to ensure coverage of all worker characteristics, ranging from the specific to the general. The 
accomplishment of this objective is enhanced by consideration of the NSSB common language 
framework to describe both work (critical work functions, key activities, performance indicators) and 
the worker (academic, employability, and occupational skills). The academic and employability skills, 
in particular, must be taught and assessed. 




-15- 



17 



Bibliography 



Altschuld, J.W., & Austin, J.T. (In press). Certification. In S. Mathison (Ed.), Encyclopedia 
of evaluation. Thousand Oaks, CA: Sage. 

American Educational Research Association, American Psychological Association, & National 
Council on Measurement in Education. (1999). Standards for educational and psychological 
testing. Washington, DC: American Educational Research Association. 

Ananda, S.M., & Rabinowitz, S.N. (1995). Developing a comprehensive occupational 
certification system. In Skill standards: Making the connections in the education and training 
network (pp. 2-24). Washington, DC: Institute for Educational Leadership. 

Bishop, J.H. (1996). Signaling the competencies of high school students to employers. In L.B. 
Resnick & J.G. Wirt (Eds.), Linking school and work: Roles for standards and assessment (pp. 79- 
124). San Francisco: Jossey-Bass. 

Browning, A.H., Bugbee, Jr., A.C., & Mullins, M.A. (Eds.). (1996). Certification: A NOCA 
handbook. Washington, DC: National Organization for Competency Assurance. 

Cameron, C.A., Beemsterboer, P.L., Johnson, L.A., Mislevy, R.J., Steinberg, L.S., & Breyer, 
F.J. (2000). A cognitive task analysis for dental hygiene. Journal of Dental Education, 64, 333-35 1 . 

Cisco Systems. (1999). Cisco networking academy program. San Jose, CA: Cisco Systems. 
Retrieved April 8, 2003, from http://cisco.netacad.nel/public/academv/index.html . 

Cizek, G.J. (Ed.) (2001). Setting performance standards: Concepts, methods, and 
perspectives. Mahwah, NJ: Erlbaum. 

Cizek, G.J., Webb, L.C., & Kalohn, J. (1995). The use of cognitive taxonomies in licensure 
and certification test development: Reasonable or customary? Evaluation in the Health Professions, 
18, 77-91. 

Council of State Governments. (1990). Occupational and professional regulation in the 
states: A comprehensive compilation. Lexington, KY : Author. 

Council on Licensure, Enforcement, and Regulation & National Organization for Competency 
Assurance. (1993). Principles of fairness: An examining guide for certification boards. Lexington, 
KY: Author. 

Council on Licensure, Enforcement, and Regulation. (1993). Development, administration, 
scoring and reporting of certification examinations: Recommendations for board members. 
Lexington, KY: Council of State Governments. 

Educational Testing Service, Inc. (2002). ETS Standards for quality and fairness. Princeton, 
NJ: Author. 

Haladyna, T.M. (Ed.). (1997). Validity of certification examinations. (Special issue). Applied 
Measurement in Education, 10, 1-104. 

Hughes, L.C., Ward, S., Grindel, C.G., Coleman, E.A., Berry, D.L., Hinds, P.S., Oleske, 
D.M., Murphy, C.M., & Frank-Stromberg, M. (2001). Relationships between certification and job 
perceptions of oncology nurses. Oncology Nursing Forum, 28, 99-106. 

Impara, J.C. (Ed.). (1995). Licensure testing: purposes, procedures, and practices. Lincoln, 
NE: Buros Institute. 

Kane, M. (1994). Validating the performance standards associated with passing scores. 

Review of Educational Research, 64, 425-461 . 




- 16 - 



Lane, S., & Stone, C. (2002). Strategies for examining the consequences of assessment and 
accountability programs. Educational Measurement: Issues & Practice, 21 (1), 23-30. 

Losh, C.L. (2000). Using national and state skill standards for vocational-technical 
education curriculum development. Columbus, OH: ERIC Clearinghouse on Adult, Career, and 
Vocational Education. 

Lyons, K J., Greening, S., & Robeson, M (2000). Final report of a study to assess the validity 
and reliability of the standards of eight allied health professions. Journal of Allied Health, 29, 41-47. 

McMillan, S.C., Heusinkveld, K., Chai, S., Murphy, C.M., & Huang, C-Y. (2002) Revising 
the blueprint for the Oncology Certified Nurse® examination: A role delineation study. Oncology 
Nursing Forum, 29, El 10-1 17 (online, n.p.). 

Mahlman, R.A., Jeong, S.B., & Austin, J.T. (2000, December). Occupational certification 
in career-technical education and employment. Presentation to the Association for 
Career-Technical Education, San Diego, CA. 

Manufacturing Skill Standards Council (2001, May). A blueprint for workforce excellence: 
Core and concentration skill standards for manufacturing. Washington, DC: Author. 

Mehrens, W.A. (1995). Le^l issues involved in licensing examinations. Educational 
Measurement: Issues & Practice, 14 (2), 16-18. 

Mehrens, W.A. (1997a). The consequences of consequential validity. Educational 
Measurement: Issues & Practice, 16 (2), 16-18. 

Mehrens, W.A. (1997b). Validating licensing and certification test score interpretations and 
decisions: A response. Applied Measurement in Education, 10, 97-104. 

Mehrens, W.A., & Popham, W.J. (1992). How to evaluate the legal defensibility of high-stakes 
tests. Applied Measurement in Education, 5, 265-283. 

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from 
persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 
50, 741-749. 

Microsoft Corporation. (2000, November 22). Redmond, WA: Microsoft Corporation. 
Retrieved November 16, 2000, from www.microsoft.com/trainimiandservices . 

National Commission for Certifying Agencies. (2003). Standards for the accreditation of 
certification programs. Washington, DC: National Organization for Competency Assurance. 

National Organization for Competency Assurance. (1999). Setting passing standards for 
performance-based certification and licensure examinations. Washington, DC: Author. 

National Organization for Competency Assurance. (2001). Certification testing on the 
Internet. Washington, DC: Author. 

National Skills Standards & Assessment Collaborative (1998, January). Cross-industry 
assessment and certification: Framework and implementation guide. San Francisco, CA: WestEd. 

National Skills Standards Board (2000a). Built to work: A common framework for skill 
standards. Washington, DC: U.S. Department of Labor. 

National Skill Standards Board. (2000b). Credentials for success. Washington, DC: Author. 

National Skills Standards Board (2001a). An introduction to the use of skill standards and 
certifications in WIA programs. Washington, DC: U.S. Department of Labor. 

National Skills Standards Board (2001b). Getting started on assessment: Developing a 
voluntary system of assessment and certification based on skill standards. Washington, DC: U.S. 
Department of Labor. 




-17- 



19 



Niebuhr, B.S., & Muenzen, P. (2001). A study of perianesthesia nursing practice: The 
foundation for newly revised CPAN and CAPA certification examinations. Journal of Perianesthesia 
Nursing, 16, 163-173. 

Plake, B.S. (1998). Setting performance standards for professional licensure and certification. 
Applied Measurement in Education, 11, 65-80. 

Rabinowitz, S.N. (1997). A comprehensive performance-based system to address work 
readiness. In H.F. O'Neil (Ed.), Workforce readiness: Competencies and assessment (pp. 

327-355). Mahwah, NJ: Erlbaum. 

Raymond, M.R. (2002). A practical guide to practice analysis for credentialing examinations. 
Educational Measurement: Issues & Practice, 21 (3), 25-37. 

Rosenbaum, J.E. (2002, April). Beyond empty promises: Policies to improve transitions 
into college and jobs. Commissioned paper for the U.S. Department of Education, Office of 
Vocational and Adult Education. Washington, DC: U.S. Department of Education, Office of Vocational 
and Adult Education. 

Sales and Service Voluntary Partnership, Inc. (2001, October). Customer service and sales 
skills standards. Washington, DC: National Skill Standards Board. 

Schmitt, K. (1995). What is licensure? In J.C. Impara (Ed.), Licensure testing: Purposes, 
practices, and procedures (pp. 3-32). Lincoln, NE: Buros Institute of Mental Measurements. 

Schmitt, K., & Shimberg, B. (1996). Demystifying occupational and professional 
regulation: Answers to questions you may have been afraid to ask. Lexington, KY: Council on 
Licensing, Enforcement, and Regulation. 

Schoon, C.G., & Smith, I.L. (Eds.) (2000). The licensure and certification mission. New 
York, NY: Professional Examination Service. 

Shimberg, B. (1981). Testing for licensure and certification. American Psychologist, 36, 
1138-1146. 

Shimberg, B., Esser, B.F., & Kruger, D.H. (1973). Occupational licensing: Practices and 
policies. Washington, DC: Public Affairs Press. 

Showers, B. (1999). Reference guide for auditing a credentialing examination program. 
Resource Brief Lexington, KY: Council on Licensing, Enforcement, and Regulation. 

Smith, I.L., & Hambleton, R.L. (1990). Content validity studies of licensing examinations. 
Educational Measurement: Issues & Practice, 9(4), 7-10. 

Tamblyn, R. (1994). Is the public being protected? Prevention of suboptimal medical practice 
through training programs and credentialing examinations. Evaluation in the Health Professions, 1 7, 
198-221,236-241. 

Violate, C., Salami, L., & Muiznieks, S. (2002). Certification exams for massage therapists: A 
psychometric analysis. Journal of Manipulative and Physiological Therapeutics, 25, 111-115. 

Zemsky, R. (1997). Skills and the economy: An employer context for understanding the 
school to work transition. In A. Lesgold, M.J. Feuer, & A.B. Black (Eds.), Transitions in work and 
learning: Implications for assessment (pp. 34-61). Washington, DC: National Academy Press. 



- 18 - 



20 



BEST COPY AVAILABLE 



1 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

Reproduction Basis 



X 



This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 



EFF-089 (1/2003) 

ERIC 



